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Kindly enter the following Preliminary Amendment in the 
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In the Claims: 

Please amend the Claims to read as follows (a copy of the 
amended claims showing the additions and deletions appears at 
the end for the Examiner's convenience): 

3 . The method as claimed in claim 1 wherein the biological 
objects to be determined are marked with one or plural chemical 
markers before the microscope images are taken. 

5. The method as claimed in claim 3 wherein said chemical 
markers are fluorochrome markers and the microscope images are 
fluorescence images . 

Please add the following new claims 14 and 15 : 

14. The method as claimed in claim 2 wherein the biological 
objects to be determined are marked with one or plural chemical 
markers before the microscope images are taken. 

15. The method as claimed in claim 4 wherein said chemical 
markers are fluorochrome markers and the microscope images are 
fluorescence images. 
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This Preliminary Amendment puts the claims into proper form 



for examination. 
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new claims 14 and 15 have been added; and claims 1, 2, 4, and 



6-13 remain unchanged 



Kindly calculate the filing fee based on 



the amended claims. 
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the present application. 
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ET AL. 
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Attorney for Applicants 
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Ten Post Office Square 
Boston, MA 02109 
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Telecopier: (617) 451-0313 
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Red- lined Claims for the Examiner s convenience : 



3 . The method as claimed in claims 1 — &r — 3- wherein the 
biological objects to be determined are marked with one or 
plural chemical markers before the microscope images are taken, 

5. The method as claimed in claims 3 or 4- wherein said chemical 
markers are fluorochrome markers and the microscope images are 
fluorescence images. 



GAGNEBIN & HAYES, 
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Nattkemper, T., et al. 
U.S. Patent Application 

5 MF.THO D FOR_THE_^UTOMA TIC ANALYSIS OF . MiaROSCOP£J[MAGES 



The invention relates to a method for the automatic 
10 analysis of microscope images of biological objects, in 

particular for the analysis of fluorescence images of 
cells . 

In numerous experiments in the field of modern 
biomedical research, fluorescence techniques, amongst 

15 others, are used for marking in order to make 

biologically significant objects visible under the 
microscope. Such objects may be cells of a certain type 
or of a certain state, for example. In recent years, 
the progress made in automating such experiments has 

20 made it possible for biomedical laboratories to perform 

large numbers of such image-producing experiments fully 
automatically . 

DE 197 09 348 CI, for example, describes a 
fluorescence microscope technique which produces a set 

25 of n images of a single sample by preparing said 

lymphocyte sample with n different fluorochrome markers. 
In each image, different lymphocyte subsets will be 
fluorescent and appear to have shining outlines. Each 
lymphocyte in the sample will have its specific 

30 fluorescent behavior in the image set. It will appear 

fluorescent in one subset of images, otherwise remaining 
invisible in the other images of the set. 
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For extracting the fluorescence patterns from the 
image set, the fluorescent lymphocytes will first of all 
have to be detected in the n images. The fluorochrome 
marked lymphocytes differ in number, location and 

5 intensity. Since such vast numbers of images and image 

data will result, from which the information will first 
of all have to be extracted for biological 
interpretation later on, there will be a so-called bot- 
tleneck in the evaluation of the experiments. Image 

10 interpretation by human employees is impractical since 

it is too time-consuming and the results are often not 
reliable. This is due to the visual evaluation work 
which is tiring and will lead to concentration losses 
already after a short time only. Moreover, the objects 

15 to be detected differ in number, location and intensity. 

Consequently, image parameters such as contrast and 
noise will differ from image to image. Furthermore, the 
objects, for example cells in tissue samples, vary 
considerably as regards their shape and size, 

20 Consequently, there is a need for automatic 

evaluation methods which are capable of locating the 
objects to be detected in one image. 

Earlier work on the automation of cell detection 
essentially focused on model-based approaches. 

25 Numbering amongst these is also the idea to adapt a 

geometric model to a gradient ensemble (Mardia et al., 
1997, In: IEEE Transactions of Pattern Analysis and 
Machine Intelligence, 19: 1035-1042). This also 

includes the exploitation of wave propagation (Hanahara 

30 and Hiyane, 1990, In: Machine Visions and Applications, 

3: 97-111) or a Hough transformation for detecting 
circular objects (Gerig and Klein, 1986, In: Proc. Int. 
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Conf. on Pattern Recognition, 8: 498-500). These 
approaches, however, have the disadvantage that they are 
frequently susceptible to changes in the shape of the 
object, and they may not be readily adapted by persons 
5 not skilled in the art. Furthermore, the images will 

not infrequently contain noise, owing to heterogeneous 
lighting conditions, and the cells will be partly 
obscure, which makes detection by boundary scanning 
unsuitable (Galbraith et al.; 1991, In: Cytometry, 12: 

10 579-596) . 

It is therefore the object of the present invention 
to provide a method of the aforementioned kind 
comprising a simple automatic cell detection procedure 
which is fast and readily adapted. 

15 This object is accomplished by a method having the 

features set out in claim 1. 

Advantageous embodiments are described in the sub- 
claims . 

The present invention relates to a method for the 
20 automatic analysis of microscope images of biological 

objects, in particular for the analysis of fluorescence 
images of cells, comprising the following steps: a) at 
least two microscope images are taken of a sample 
containing a plurality of biological objects; b) a first 
25 microscope image is selected and the positions of mass 

gravity centers of a number n of the individual objects 
visible in said first microscope image are marked, in 
which process each marked object is assigned a defined 
first image excerpt completely surrounding the marked 
30 object, and in which each first image excerpt containing 

a marked object is assigned the value 1, with the number 
n of such marked first image excerpts constituting a 
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positive training set; c) a number m of second image 
excerpts which are spaced a predetermined minimum 
distance from the first image excerpts are selected and 
marked, with the size and shape of a second image 
5 excerpt corresponding to the size and shape of a first 

image excerpt, in which process each second image 
excerpt is assigned the value 0, with the number m of 
such marked second image excerpts constituting a 
negative training set; d) characteristic features and/or 

10 feature combinations of the positive and negative 

training sets are determined and assigned to a 
classification value between 0 and 1, said 
classification value representing the degree of 
probability of the presence of a marked object, and the 

15 determined features and/or feature combinations are 

stored; e) classification values of all image points of 
the second and each further microscope image are 
automatically determined by comparing the image data of 
the second and each further microscope image with the 

20 features and/or feature combinations determined in 

procedural step d) , in which process the classification 
value for an image excerpt surrounding the image point 
is determined for each image point of the second and 
each further microscope image, said image excerpt 

25 corresponding in size and shape to said first or second 

image excerpt; and f) the position (s) of biological 
objects in the second or each further microscope image 
is/are recognized by evaluating classification values, 
said determined classification values being then 

30 compared with a predetermined threshold value which 

represents the presence of a biological object. 
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The method of the invention for detecting 
biological objects such as cells in microscope images 
advantageously uses classification values which will 
assign to image excerpts of given size and shape a value 
5 between 0 and 1 and which will indicate whether (1) or 

not (0) a biological object can be seen in this image 
excerpt. However, it is also possible to use any other 
numerical values which are different from each other. 
In an image, all cells are found by assigning all image 

10 areas to one classification value. A simple automatic 

search for high values (threshold value analysis) will 
thus allow the positions of the biological objects to be 
located automatically and stored in a file and processed 
further, for example. Providing such classification 

15 values, or a similar classifier, and simply adapting 

them in case of significant changes in the microscope 
images due to microscope modifications, different 
samples, different fluorescence markers or different 
cell types is made possible by the use of the method of 

20 the invention which constitutes an artificial neural 

network. 

The method of the invention will allow a fully 
automatic detection of biological objects such as cells 
in microscope images, after it has been trained by a 

25 user by being provided with a set of cell-positive image 

excerpts. It is particularly easy to use since it 
merely requires the user to mark cells on the screen for 
training, which is a routine activity in laboratory 
work. The system is thus suitable for the fast 

30 evaluation of numerous microscope images for localizing 

cells. Owing to its ease of use, the method of the 
invention can be readily applied and executed. It is 
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suitable for supporting any kind of empirical work with 
microscope images of biological objects at universities 
as well as in industry - which sectors have a vast 
throughput of samples and require objective and 
5 reproducible evaluations. 

In a further advantageous embodiment of the method 
of the invention, the sample is a tissue sample and the 
biological object is a cell. In particular for 
automatic cell detection, the method of the invention 

10 has shown to be advantageous over conventional methods 

of the prior art. 

In yet another advantageous embodiment of the 
method of the invention, the biological objects to be 
determined are marked with one or plural chemical 

15 markers before the microscope images are taken. In this 

process, a bleaching or rinsing operation may be 
performed between the taking of the individual 
microscope images. The chemical markers may be 

fluorochrome markers and the microscope images may be 

20 fluorescence images. 

For example, fluorochrome markers are used on 
lymphocytes in order to determine the presence of 
proteins in lymphocyte cell surfaces. Each marker will 
bind to a lymphocyte subset which depends on the 

25 existence of the respective protein in the lymphocyte 

surface membrane. Fluorescence excitation will cause 
the binding lymphocytes to appear with high intensities. 
A corresponding microscope image will be taken by a CCD 
camera. The marker will subsequently be removed from 

30 the cells by means of bleach, and the operation will be 

repeated using a different marker. In the embodiment 
described here the process comprising marking, image 



generation and bleaching can be repeated with up to nine 
markers. During each repetition, the lymphocyte 

positions will remain unchanged, which allows a matching 
or coincidence comparison of the positions in the 
5 different images. This experimental setup is especially 

used for analyzing T lymphocytes which have invaded the 
muscular tissue in a clinical case of sarcoidosis. The 
lymphocytes were prepared with n=7 markers, and seven 
fluorescence images were taken, in which different 

10 subsets of the cells appeared fluorescent. On 

conclusion of all n steps, a set of n microscope images 
with different lymphocyte subsets can be evaluated. The 
n microscope images are evaluated by means of the method 
of the invention. 

15 In this case, the number n of the individual 

biological objects, in particular cells, marked in 
procedural step b) will be larger than or equal to 50. 
For each marked point, it will be decided whether or not 
such point constitutes a cell center. Since the 

20 lymphocytes, for example, will be in organic fiber 

material during an invasion situation, they will not be 
circular in shape, but irregularly oblong, which shape 
is referred to as predominantly convex. For the 

classification of the biological objects, a special form 

25 of a neural network, the so-called "Local Linear Map" 

(LLM) (Ritter, 1991: In: Artificial Neural Networks, 
Elsevier Science Publishers B.V.) is used. The LLM 
classifier is trained by a set of image excerpts which 
contain cells. For training the LLM classifier, 

30 according to procedural step b) an image is selected 

from the image set. Furthermore, e.g. by means of a 
computer mouse, a group of fluorescing cells are marked. 



The group of image excerpts dimensioned N x N 
surrounding these cells will constitute the set of 
positive training examples or a positive training set. 
The first image excerpt is sguare in shape, according to 
5 an advantageous embodiment of the method of the 

invention, with the size N x N or the side length N of 
the first image excerpt corresponding to the maximum 
diameter of the biological objects in the first 
microscope image. 

10 A further set of second image excerpts is randomly- 

selected fully automatically from the same image, 
according to procedural step c) , thereby keeping a 
minimum distance of e.g. 3-5 pixels from the already 
marked first image excerpts. This second set constitutes 

15 the set of negative training examples or a negative 

training set. According to an advantageous embodiment 
of the invention, the number m of second image excerpts 
will be larger than or equal to 50, with the second 
image excerpts being defined automatically, at the same 

20 time keeping to the minimum distance from the respective 

first image excerpts. 

In an embodiment, a din dimensional feature vector x 
will be calculated for each training example excerpt. 
The set of selected image excerpts will be subjected to 

25 a principal component analysis (PCA) . This is a known 

technique for classification tasks in computer vision. 
The basic idea of PCA is that the high-dimensional image 
excerpt is mapped to a considerably smaller dimensional 
(din = approx. 6) feature space. These features will 

30 give so-called input feature vectors x. The feature 

vectors have clearly less data than the actual image 
excerpts. Calculation of the feature vectors for the 
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positive and negative input examples will result in the 
training set of the (input, output) pair 

r = { (Xa, Ya) }a. 

For the positive training set, ya will be set to 
equal 1, and to equal 0 for the negative ones. 

The LLM is defined by 

{Wi^"" G R^in, Wi°"^ G R^outr Ai e R^in^^out, i = 1...1}. 

A triple vi = (Wi^", Wi°"^, Ai) is referred to as a 
node. For training the LLM, a pair (Xa, ya) is randomly 
selected from F, and the learning rules 

AWk^'' = G'-'^(Xa"Wk^") (1) 

Awk°"^ = e°"^ (ya-y (Xa) ) +AkAwk^" (2 ) 



(Xa-Wk ) 

AAk = e^(ya-y(xa) ) (3) 

Ilx.-Wk-Ih 



will be implemented, e^"^, e*^^^, cz]0,l[ are descending 

learning step sizes and k is true for k = arg mink{Ilx- 
Wk^"l I }. Therefore, Wk^" is the nearest neighbor to input 
X. This is repeated 1*10,000 times. 
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The trained LLM classifier performs a mapping of 
fluorescence image points to classification or evidence 
values in (0;1). For calculating the classification 
value for e.g. a fluorescent cell at an image point, the 
5 feature vector x for its surrounding area will be 

calculated. The LLM output for the input x is 
calculated by 

y(x) = Wk°"^ X Ak(x - wi,'") (4) 
10 with k = arg mink{| I x-w/"^ | | } 

In the embodiment described, the number of nodes is 
i = 5, and the image excerpt size is N=15 pixels. 

For the detection of all biological objects, e.g. 

15 fluorescent cells, in the second or each further image 

of the image set, each image point is mapped to its 
classification value by equation (4). The surrounding 
image region of the point will be supplied to the 
classifier which will then calculate its classification 

20 value. By calculating the classification values for 

each point in an image, a so-called classification or 
evidence map of the image will be obtained. Regions of 
high classification values or evidences will indicate 
biological objects such as fluorescent cells in the 

25 fluorescence image in question. All points of a 

classification value higher than e.g. 0.5 and without 
major evidences in their vicinity will form the set of 
positions of biological objects such as fluorescent 
cells in the image. A similar approach will be adopted 

30 for all further microscope images of the image set. 

In accordance with an advantageous embodiment of 
the invention, the classification values of all image 
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points of the second and each further microscope image 
are determined automatically according to procedural 
step e) by scanning the image surface of the second and 
each further microscope image. 

5 In another advantageous embodiment of the method of 

the invention, the object positions determined by the 
procedural steps a) to f) are compared in the totality 
of the microscope images - which will determine the 
spatial position and the distribution of the individual 

10 objects in the sample. 

This will allow fluorescence patterns of cells, for 
example, to be determined. In doing so, after the cell 
detection process in all images, locally corresponding 
fluorescence sites are found in different images so as 

15 to map fluorescences of the same cell to its marker 

combination pattern. This coincidence analysis is 
exclusively based on the positions of the cells in the 
images, and is a difficult task since the determined 
positions of one and the same cell will not always be 

20 exactly identical in different images. Furthermore, the 

number of fluorescent cells in the images will vary 
strongly. Both aspects make a simple matching on the 
basis of exclusively the detected cell positions 
impossible. The method of the invention avoids such 

25 problems by evaluating the classification values or 

evidences of all images of the image set. 

By determining the classification values or 
evidences of all n images as described above, one will 
obtain n classification or evidence values for each 

30 point. Subsequently, the maximum evidence value for 

each point will be selected and entered into a new 
evidence map at its coordinates. This map is referred 
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to as "Master Evidence Map" since each biological 
object, for example a cell, which occurred in at least 
one of the images, will be represented by a high 
evidence value in this map. Applying procedural step f) 

5 to said "Master Evidence Map" will result in all 

positions of the biological objects. This set of M cell 
positions { (xi, yi) } is referred to as the "master set" . 
Subsequently, the binary fluorescence values fj'""' will 
be collected for each biological object from the "master 

10 set". fj of Pi =(fi*^^, fn'^') will be set to 1 when 

a biological object such as a cell in the j-th fluores- 
cence image of the image set was detected in close 
vicinity to its coordinates {x±, yi) . A simple local 
matching procedure between the master set and the 

15 classification values of the positions of the biological 

objects, or cell positions, detected for all 
fluorescence images will give the binary fluorescence 
patterns of all detected objects or cells. 

One use, according to the invention, of the 

20 described new method for the automatic analysis of 

microscope images of biological objects is the automatic 
cell classification of fluorescent cells. For example, 
a set of seven fluorescence images was taken according 
to the method of the invention. The seven different 

25 markers are cd2, cd3, cd4, cd8 , cdllb, cdl9 and cd26; 

these are antibody markers commonly used in fluorescence 
microscopy. The training set of all cell excerpts was 
selected by hand, from the cd4 image, to be precise. In 
each image, the fluorescent cells were detected by means 

30 of the LLM classifier. Using the maximum condition of 

the classification values or evidences calculated by the 
LLM, the "Master Evidence Map" was generated. The 
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positions of M = 550 fluorescent cells were then 
extracted from said "Master Evidence Map" . Finally, the 
local matching step produced the marker combination 
pattern pj , j = 1, 550, 
5 For examining the distribution of the marker 

combination patterns within the lymphocyte group, their 
binary patterns (fi'""', f?*""') were mapped from the 

dual system to a numerical label in the decimal system. 
The pattern frequencies were counted and illustrated in 

10 the form of a histogram (histogram including 7? = 128 

bars ) . This has shown that only 24 of 128 possible 
patterns in the total lymphocyte group were found. 
Three patterns dominate in number, (1000000) (= 1) , 
(0010000) (= 8) and (1010000) (= 9). These are cells 

15 which bound to only cd2 or cd4, or to both these markers 

only. The remaining frequencies are below 30. 

In order to get an idea of the coincidence of a 
certain marker compared to the remaining markers, a 
corresponding histogram was calculated for each marker. 

20 The absolute number of cells which were made fluorescent 

by this selected marker were shown. It was possible to 
determine that the absolute number of fluorescent cells 
will vary strongly. Most dominant are the markers cd2 
and cd4 . Lymphocytes fluorescence-marked with cdl9 

25 occur seldom and only once coincide with another marker 

(cd2) . This means that this marker is highly selective. 
A strong coincidence may also be observed for example 
between the pairs (cd2, cdS), (cd3, cd8) and (cd2, cdS) . 

In the embodiment described, microscope images of a 

30 single visual field were recorded and analyzed. 

However, if one is confronted with an enormous number of 
several hundred visual images, for example, then the 
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method of the invention will allow a precise statistical 
analysis of those immune cell subsets which have 
penetrated the tissue sites, within a relatively short 
time . 
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What is claimed is: 

1. A method for the automatic analysis of microscope 
images of biological objects, in particular for the 
5 analysis of fluorescence images of cells, comprising the 

following steps: 

a) taking at least two microscope images of a sample 
including a plurality of biological objects; 

b) selecting a first microscope image and marking the 
10 position (s) of mass gravity centers of a number n 

of the individual objects discernible in the first 
microscope image, in which step each marked object 
is assigned a defined first image excerpt which 
completely surrounds the marked object, and each 
15 first image excerpt including a marked object is 

assigned the value 1, with the number n of such 
marked first image excerpts constituting a 
positive training set; 

c) selecting and marking a number m of second image 
20 excerpts each spaced a predetermined minimum 

distance from said first image excerpts, with a 
second image excerpt corresponding in size and 
shape to said first image excerpt, in which step 
each second image excerpt is assigned the value 0, 
25 with the number m of such marked second image 

excerpts constituting a negative training set; 

d) determine characteristic features and/or feature 
combinations of the positive and negative training 
sets and assigning said characteristic features 

30 and/or feature combinations to a classification 

value between 0 and 1, said classification value 
representing the degree of probability of the 
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presence of a marked object, and the determined 
features and/or feature combinations are stored; 
e) determine classification values of all image 
points of the second and each further microscope 
5 image by comparing the image data of the second 

and each further microscope image with the 
features and/or feature combinations determined in 
procedural step d) , in which step, for each image 
point of the second and each further microscope 

10 image, the classification value for an image 

excerpt surrounding the image point is determined 
and the size and shape of this image excerpt 
corresponds to the size and shape of the first or 
second image excerpt; and 

15 f) recognizing the position (s) of biological objects 

in the second or each further microscope image by 
evaluating the determined classification values, 
in which step the determined classification values 
are compared with a given threshold value 

20 representing the presence of a biological object. 

2 . The method as claimed in claim 1 wherein the sample 
is a tissue sample and the biological object is a cell. 

25 3. The method as claimed in claims 1 or 2 wherein the 

biological objects to be determined are marked with one 
or plural chemical markers before the microscope images 
are taken. 

30 4. The method as claimed in claim 3 wherein the objects 

to be determined are marked with one or plural chemical 
markers before the microscope images are taken, with a 
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bleaching or rinsing procedure being performed between 
the taking of the individual microscope images. 

5. The method as claimed in claims 3 or 4 wherein said 
5 chemical markers are fluorochrome markers and the 

microscope images are fluorescence images. 

6. The method as claimed in claim 1 wherein the 
microscope images are taken by a CCD camera and then 

10 digitized. 

7. The method as claimed in claim 1 wherein the number n 
of the individual biological objects marked in 
procedural step b) is larger than or equal to 50. 

15 

8. The method as claimed in claim 1 wherein the first 
image excerpt is of square shape, with the size and/or 
side length of the first image excerpt corresponding at 
least to the maximum diameter of the biological objects 

20 in the first microscope image. 

9. The method as claimed in claim 1 wherein the number n 
of second image excerpts is larger than or equal to 50, 
with the second image excerpts being defined 

25 automatically, keeping to the minimum distance from the 

respective first image excerpts. 

10. The method as claimed in claim 1 wherein 
classification values of all image points of the second 

30 and each further microscope image are automatically 
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determined according to procedural step e) by scanning 
the image surface of the second and each further 
microscope image. 

5 11. The method as claimed in claim 1 wherein the 

threshold value of the classification value representing 
the presence of a biological object is at least 0.5. 

12. The method as claimed in claim 1 wherein the object 
10 positions determined by procedural steps a) to f) are 

compared in the total number of microscope images so as 
to obtain a spatial location and distribution of the 
individual objects in the sample. 

15 13. Use of a method as claimed in claim 1 for the 

automatic cell classification of fluorescent cells. 
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